A Study of “Irrelevant” Items: Impact on Bookmark Placement and Implications for College and Career Readiness

The judgmental standard setting studies for 12 Grade NAEP Preparedness Research are the focus of this paper. While the standard setting methodology was very familiar, the postsecondary context of standard setting was new to the Governing Board. This was the first effort to set a cut score on NAEP to represent minimal academic preparedness, and the studies were to be for both college course placement and entry to job training programs. The Governing Board made no assumptions regarding the relationship between preparedness for college course work and the job training programs; the Board wanted to know if the academic knowledge and skills required to be minimally prepared for “college and career” are similar. Clearly, there was much to be learned about standard setting in this new post-secondary context, and this paper reports on the issues surrounding “irrelevant items” in a standard setting context. The potential for a mismatch of academic knowledge and expertise on the part of occupational job trainers with that measured by the grade 12 National Assessment of Educational Progress (NAEP) was anticipated, especially with respect to the grade 12 mathematics NAEP. The compilation of results from the studies implemented across a two-month period pointed to an ultimate conclusion that there should be no reference points on NAEP associated with the outcomes of these studies. The results of the judgmental standard setting studies simply did not confirm nor were they confirmed by results from other studies. Gathering more information about the judgments and potential impacts on the judgments that were the ultimate outcomes of the process was a growing research priority. A special study was designed and implemented to collect more information about the issue of irrelevant items and the potential impact of these items on the judgments of panelists. Introduction and Background for this Research The need for U.S. students to be academically prepared for college and careers seems clear to everyone from the President to the postman. College and career readiness research is at the forefront of major large-scale assessment programs at the national and state level in this country. The Common Core State Standards call for measures and standards of college and career readiness; ACT, Inc. has college and workplace readiness benchmarks for their assessments to predict college course grades; the College Board has developed benchmarks to predict freshman year grade point averages; and the National Assessment Governing Board has launched an 1 I wish to acknowledge the assistance of Dr. Luz Bay and Mr. Chris Clough of Measured Progress in preparation of this paper. They provided data collected in the special study for use in writing this paper. 2 This paper was prepared for presentation in the session Setting Academic Preparedness Standards for Job Training Programs: Are We Prepared? at the annual meetings of the National Council on Measurement in Education, Vancouver, British Columbia, Canada, April 14, 2012. More information about the 12 Grade NAEP Preparedness Research is available at www.nagb.org.